Faults will frequently occur during the computational process of the hardware based SMS4 algorithm. The attacker can easily break the algorithm by using the fault information and performing the fault attack. In order to solve this issue, a new fault detection method for SMS4 was proposed. Firstly, locations of the fault occurrence and the impact of the faults were analyzed. Then, three detection position points on the critical path were targeted, and by monitoring these three points in real-time to locate the faults. Once a fault was successfully detected, the system would immediately re-execute the algorithm to avoid the attacker obtaining the fault information. Furthermore, the proposed SMS4 with fault detection and the original SMS4 without fault detection were implemented on two Field Programmable Gate Array (FPGA) platforms respectively, including Virtex-7 of Xilinx and Cyclone Ⅱ of Altera. Compared with the original SMS4, hardware resource of the proposed SMS4 with fault detection was increased by 30% with similar throughput on Virtex-7. Hardware resource of the proposed SMS4 with fault detection was increased by 0.1% and the throughput was around 93% on EP2C35F76C6. The experimental results show that the proposed algorithm can effectively detect faults using affordable hardware resource to avoid fault attack without affecting throughput.